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ABSTRACT 


A  simple  mathematical  formalism  is  presented  suggesting  a 
mechanism  for  computing  relative  depth  of  any  two  texture  elements 
characterized  by  the  same  relative  motion  parameters.  The  method 
is  based  on  a  ratio  of  a  function  of  t*e  angular  velocities  of 
the  projecting  rays  corresponding  to  the  two  texture  elements. 

The  angular  velocity  of  a  ray  cannot,  however,  be  computed  direct¬ 
ly  from  the  instantaneous  characterization  of  motion  of  a 
"retinal"  point.  It  is  shown  hew  it  can  be  obtained  from  the 
(linear)  velocity  of  the  image  element  on  the  projection  surface 
and  the  first  time  derivative  of  its  direction  vector.  A  similar 
analysis  produces  a  set  of  equations  which  directly  yield  local 
surface  orientation  relative  to  a  given  visual  direction.  The 
variables  involved  are  scalar  quantities  directly  measurable  on 
the  projection  surface  but,  unlike  the  casf  of  relative  depth, 
the  direction  of  (instantaneous)  motion  has  to  be  computed  by 
different  means  before  the  method  can  be  applied.  The  relative 
merits  of  the  two  formalisms  are  briefly  discussed. 
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1.  Introduction 

Optical  flow  is  the  distribution  of  angular  velocities  of  pro¬ 
jecting  rays  due  to  relative  motion  of  objects  with  respect  to 
the  observer.  Conceptually,  optical  flows  undoubtedly  carry  a 
wealth  of  information  about  the  spatial  arrangement  of  the 
viewed  scene,  and  prominent  psychologists  such  as  Gibson  (1950, 
1979)  have  argued  forcefully  for  the  predominant  role  of  this 
information  in  human  vision.  Discontinuities  in  the  distribution 
of  angular  velocities  have  been  shown  to  directly  correspond  to 
occluding  (or  self-occluding)  edges  (Nakayama  and  Loomis,  1974) . 
Corresponding  discontinuities  in  "retinal"  motions  thus  offer 
powerful  information  for  segmentation  purposes.  Some  recent  work 
has  illuminated  some  of  the  relationships  between  variables 
directly  involved  in  the  formation  of  optical  flows  (Koenderink  and 
van  Doom,  1975,  1976,  1977;  Longuet-Higgins  and  Prazdny,  1980; 
Prazdny,  1980) . 

The  purpose  of  this  paper  is  to  present  a  mathematical  analy¬ 
sis  of  some  relations  containing  information  about  spatial  dis¬ 
positions  of  a  set  of  texture  elements.  Using  the  concept  of 
polar  projection  as  the  model  for  the  physical  image-forming 
process  we  show  that  the  relative  depth  of  two  texture  elements 
can  be  computed  as  a  simple  ratio.  The  entities  involved  are  the 
angular  velocities  of  the  rays  through  the  texture  elements  and 
the  center  of  projection,  and  the  visual  directions  of  the  rays, 
which  are  unit  vectors  specifying  the  directions  of  the  rays  in 


some  egocentric  reference  frame  centered  at  the  center  of  pro¬ 
jection.  We  show  that  the  angular  velocity  at  an  image  loca¬ 
tion  can  be  obtained  from  the  image  velocity  vector  and  its  first 
time  derivative  at  that  locus. 

A  slightly  different  analysis  requiring  an  a  priori  know¬ 
ledge  of  the  direction  vector  of  the  translatory  component  of 
the  relative  motion  leads  to  an  interesting  characterization  of 


Underlying  our  research  on  the  interpretation  of  image 
motions  is  the  assumption  that  (an  approximation  to)  the  velocity 
vectors  associated  with  individual  image  elements  can  be  obtained 
with  reasonable  accuracy.  Some  recent  research  regarding  the 
computation  of  "retinal"  velocities  directly  from  the  image 
brightness  values  (Hadani  et  al.  1980;  Horn  and  Schunck,  1980) 
supports  this  arsumption.  Other  promising  support  comes  from  the 
research  on  discrete  solutions  to  the  correspondence  problem 
(Ullman,  1979;  Barnard  and  Thompson,  1980)  which  relies  on  match¬ 
ing  various  higher-level  image  "token"  structures. 
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2.  The  basic  image  forming  geometry 

To  begin,  let  us  first  consider  motions  of  the  projecting 
rays  independently  of  any  particular  projection  surface.  Refer 
to  Figure  1.  A  texture  element  P  projects,  along  the  ray  OP, 
into  the  image  element  Q  on  the  unit  sphere  centered  at  the 
center  of  projection,  0.  As  the  object  moves  relative  to  the 
observer,  the  ray  OP  (instantaneously)  rotates  about  an  axis 
through  0  causing  the  image  element  Q  to  trace  a  path  (T3)  on 
the  unit  sphere. 

Consider  a  unit  vector  (Q)  determining  the  direction  of  the 
ray  OP  at  a  given  instant  <Note  1>.  Its  velocity  is  d/dt(Q)=Q. 

Q  is  perpendicular  to  Q.  As  the  ray  moves  with  angular  velocity  A, 
Q  moves  along  T3  with  (linear)  velocity  Q  (=v').  The  two  velo- 
cities  are  related  by  Q=AxQ.  Later  we  will  show  how  the  velocity 
v"  of  an  image  element  on  the  projection  plane  relates  to  the 
velocity  Q  of  the  unit  vector  Q.  For  the  moment  it  suffices  to 
note  that  the  equation  Q=AxQ  does  not  determine  A  uniquely;  it 
only  constrains  A  to  lie  in  the  plane  a  normal  of  which  is  Q 
(see  Figure  8  for  further  explanation) . 

The  motions  of  the  object  and/or  the  observer  are  relative, 
i.e.,  we  can  only  resolve  the  motion  of  the  object  relative  to 
the  observer  (or  vice  versa) .  The  (instantaneous)  motion  of  an 


object  with  respect  to  the  observer  (in  the  reference  frame  in 
which  the  observer  is  stationary)  can  always  be  described  as  a 
rotation  (with  some  angular  velocity  AR)  superimposed  on  a  trans¬ 
lation  specified  by  a  vector  v.  The  axis  of  rotation  can  be 


chosen,  without  loss  of  generality,  to  pass  through  the  center 
of  projection  (Chasles*  theorem)  to  remove  the  ambiguity 
(Whittaker,  1944).  The  total  (linear)  velocity  of  an  environ¬ 
mental  point  P  (with  position  vector  P)  is  then  V=v+A*P.  An 
equivalent  expression  in  terms  of  the  angular  velocity  of  the 
projecting  ray  OP  is 
(1)  A=A^+Ar 

where  A^  is  the  angular  velocity  due  to  the  translation  alone, 
and  Ar  is  the  rotational  component  (note  that  the  values  of  AR 
and  v  specify  relative  motion,  and  not,  in  any  sense,  the  actual 
3D  motion  of  the  object) .  The  simplicity  of  equation  (1)  results 
from  the  fact  that  angular  velocities  about  a  common  point  add 
vectorially  (Weatherburn,  1965) .  Observe  that  AR  does  not  vary  from 
point  to  point  on  a  rigid  body;  it  is  a  property  of  the  body  as 
a  whole  and  independent  of  distance  or  visual  direction.  on 

the  other  hand,  is  a  function  of  visual  direction  and  the  distance 
of  the  texture  element  to  the  center  of  projection  (see  below) . 

Of  these  two  component  fields,  only  A^  carries  information  about 
relative  distance.  We  will  not  derive  an  expression  for  A^  here. 

The  reader  is  referred,  for  example,  to  Nakayama  and  Loomis  (1974) 
or  Prazdny  (1980)  for  a  detailed  discussion.  Briefly,  when  the 
object  translates  relative  to  the  observer,  individual  rays  of 
projection  all  move  in  the  same  plane  (different  for  different 
rays)  .  The  direction  of  A,j,  is  normal  to  this  plane  spanned  by 
the  vectors  v  and  Q.  The  magnitude  of  A^  is  equal  to  dg/dt  where 
6  is  the  angle  between  the  direction  of  translation  and  the  given 


ray  (see  Figure  2) .  is  then  given  by 

(2)  A^-de/dt  ( vxQ) /sin ( 0 ) =v/S  (vxQ) 

where  S=|op|  is  the  distance  of  a  given  texture  element  to  the 
center  of  projection. 

The  observation  that  enables  us  to  derive  an  expression  for 
the  relative  depth  of  any  two  texture  points  moving  in  the  same 
way  relative  to  the  observer  concerns  equation  (1) .  Consider  any 
two  points  P^,  Pj  on  the  same  object  <Note  2>.  From  equation 
(1)  it  follows  that 

(3)  Ai-^pi+AR  and  A.-A^+A* 

We  see  that  because  A_  is  the  same  for  the  two  points  it  cancels 
out  when  the  angular  velocities  are  subtracted: 

<4)  Ai-fArV^Ti-^Tj 

Using  (2)  and  substituting  we  obtain 

(5)  Aij=vx(k.Qi-k..Qj) 

where  k^v/S^.  We  form  the  scalar  product  of  both  sides  of  (4) 
with  (k^Q^-kjQj)  to  obtain 

(6)  k. (A. . .Q . ) -k , (A. . .Q . )=0 

i  —l]  i  ]  -i]  ] 

This  is  because  the  scalar  triple  product  involving  one  vector 
twice  is  always  zero.  We  set  a^=A^...Q^  and  substitute  back  into 


(7)  a./a^S./Sj 

In  other  words,  the  relative  depth  of  any  two  points  having  the 
same  relative  motion  parameters  (v  and  AR)  is  computable  as  a 
simple  ratio.  Observe  that  (6)  does  not  involve  AR  or  v;  relative 


depth  can  be  computed  independently  of  relative  motion.  This  is 


a  simple  but  important  finding.  In  general,  all  mathematical  for 
malisms  for  computing  surface  orientation  or  3D  structure  from 
motions  on  the  projection  surface  depend  (implicitly  or  explicit¬ 
ly)  on  computing  the  rotational  component  of  the  relative  motion 
first  (e.g.,  Ullman,  1979;  Longuet-Higgins  and  Prazdny,  1980; 
Prazdny,  1980) .  To  obtain  ar,  one  has  to  solve  a  set  of  non¬ 
linear  equations  whose  coefficients  are  the  velocity  vector  compo 
nents  of  at  least  five  neighboring  image  elements  (Prazdny,  1980) 
or  use  the  first  and  second  spatial  derivatives  of  the  image  velo 
city  field  to  obtain  enough  information  to  solve  for  AR  and  v 
directly  as  an  integral  step  in  computing  the  local  surface  orien 
tation  (Longuet-Higgins  and  Prazdny,  1980) .  Ullman' s  scheme 
(Ullman,  1979),  which  uses  orthogonal  projection  and  relies  on  a 
theorem  from  affine  geometry  (the  structure-f rom-motion  theorem 
<Note  3>)  ,  uses  not  only  spatial  information  (mutual  position  of 
a  set  of  image  elements)  but  also  temporal  information  (the  rela¬ 
tive  position  of  a  given  image  element  in  successive  snapshots) 
to  recover  the  rotation  of  the  configuration  prior  to  the  compu¬ 
tation  of  relative  depth  [see  Meiri  (1980)  for  some  comments  on 
the  number  of  image  points  and  snapshots  necessary  to  solve  for 
the  relative  motion  parameters] .  Of  course,  the  assumption  of 
orthogonal  projection  works  only  for  some  situations. 

In  contrast,  equation  (6)  above  only  requires  that  the 
angular  velocities  (A^,A^)  at  two  visual  directions  (Q^,Qj)  be 
known.  Unfortunately  the  angular  velocity  at  a  "retinal"  locus 
cannot  be  computed  from  the  information  available  at  that  locus 


at  an  instant.  The  equation  v'=AxQ  does  not  specify  A  uniquely 
(see  also  Figure  8) .  The  angular  velocity  of  a  ray  through  an 
image  point  can  be  obtained  only  when  some  additional  information 
is  available.  We  show  that  it  can  be  computed  when  the  vector 
specifying  the  time  rate  of  change  in  the  direction  of  the  "retinal" 
velocity  is  available. 

To  see  this  consider  Figure  3.  Due  to  (relative)  motion  of 
a  texture  element,  the  ray  specified  by  the  direction  vector  Q 
rotates  about  0  so  that  Q  moves  on  the  surface  of  the  unit  sphere 
with  some  velocity  v'=Q.  To  find  the  instantaneous  plane  of  rota¬ 
tion  of  Q  it  is  sufficient  to  apply  a  few  concepts  from  elementary 
differential  geometry.  Observe  that  v',  the  unit  vector  in  the 
direction  of  Q,  is  the  unit  tangent  to  the  path  at  Q.  This  means 
that  v'  is  in  the  direction  of  the  principal  normal  at  Q.  Toge- 

ther,  v'  and  v'  span  the  plane  on  which  lies  the  circle  of  curva- 

• 

ture  at  Q.  In  other  words,  the  plane  a  normal  of  which  is  v'xv' 
(this  vector  lies  in  the  direction  of  the  binormal  vector  at  Q) 
is  the  plane  of  instantaneous  rotation  of  Q,  and  A,  the  angular 
velocity  vector  of  Q,  must  lie  in  the  direction  of  this  vector. 
Observe  that  here  we  bring  in  temporal  information  to  obtain  the 
angular  velocity  vector.  Other  kinds  of  additional  information  are 
possible.  In  the  next  section  we  show,  for  completeness,  how  v' 
and  v'  relate  to  "retinal"  variables  when  the  projection  surface 
is  a  plane.  Then  we  analyze  a  method  for  computing  local  surface 
orientation  directly,  without  computing  the  relative  depth  map 


first. 


3 .  Computing  the  angular  velocity  of  a  projecting  ray  from 

"retinal"  variables 

In  this  section  we  assume  that  the  projection  surface  is  a 
plane  at  unit  distance  from  0.  As  the  projecting  ray  rotates 
about  0  with  some  angular  velocity  A,  Q  moves  with  velocity  v' 
along  T3,  and  Q=QQ,  the  point  at  which  the  ray  pierces  the  pro¬ 
jection  plane,  moves  with  velocity  v"  along  T2  (see  Figure  4) . 
Observe  also  that  T3  is  a  perspective  transformation  of  T2  and  vice 
versa.  For  example,  if  T3  is  a  circle,  T2  could  be  any  conic 
section.  The  exact  type  of  the  curve  will  depend  on  the  mutual 
orientation  of  the  plane  containing  T3  (determined  by  the  direc¬ 
tion  of  the  angular  velocity  vector  A)  and  PP.  As  mentioned 
above,  the  angular  velocity  vector  A  is  normal  to  the  instantane¬ 
ous  plane  of  rotation  of  the  ray  OP,  i.e.,  parallel  to  the  binor¬ 
mal  vector.  Our  first  task  is  thus  to  determine  the  direction  of 
the  binormal  vector  associated  with  the  motion  of  Q.  First,  we 
express  Q=v'  in  terms  of  the  velocity  of  the  image  element  at  a 
given  point,  and  then  we  will  find  the  direction  of  A  as  the 
vector  product  of  v'  and  v ' .  We  will  establish  a  few  interesting 
auxiliary  relations  before  proceeding  further. 

Consider  Figure  5.  The  figure  illustrates  the  fact  that  the 
projection  of  a  segment  with  magnitude  v’  along  a  perpendicular 
to  a  given  line  i  on  another  line  which  makes  an  angle  X  with 
line  l  is  v"=v'Q/(sin(X) -v'cos (X) ) .  We  will  refer  to  this  equation 
as  the  "radial  projection  equation."  Next  we  will  establish  a 

r. _ _  r  *  rprising  fact  about  the  relation  between  the  velocity  of 

a  point  and  the  velocity  of  its  projection.  We  will  first 


consider,  for  simplicity,  only  planar  motions,  but  the  relation 
holds  for  space  motions  too  (see  below).  Consider  Figure  6. 

Suppose  that  the  point  'a'  moves  along  a  circular  trajectory  C 
so  that  its  distance  to  0  remains  unchanged.  The  (infinitesi¬ 
mal)  displacement  is  d<p.  The  displacement  of  the  projection  of 
•a*  on  X. ^  is  tan(dcp).  The  displacement  of  the  projection  of  'a' 
on  i2  is  Qtan(d<P)/[sin(A)-cos(A)  tan(d<P)  ]  (using  the  radial  pro¬ 
jection  equation) .  To  compute  the  relation  between  the  velo¬ 
cities  on  C  and  i 2,  we  divide  by  dt  and  take  the  limit  as  dt-*-0: 

lsin(A)  -cos  (A)  tan(dcp)  dt J  sin  (A) 

This  is  because  9(t)  is  a  continuous  function  of  t,  and  lim[tan 
(x)/x]=l  as  x+0.  This  means  that  when  a  point  'a'  moves  along 
a  path  with  speed  v',  its  projection  on  the  line  %2  moves  with 
speed - 

(8)  v"=Qv'/sin(A) 

In  t>ther  words,  the  velocity  of  the  projection  of  a  point  moving 
valong  a  curve' is  not  the  projection  of  the  velocity  with  which 
the  point  moves  along  that  curve  "iNote  10>.  This  relation  holds 
also  in  3D  space  <Note  4<>.  Next  we  will  derive  the  equation  which 
will  enable  us  to  express  the  angular  velocity  in  terms  of  the 
image  velocities. 

Refer  to  Figure  4.  As  Q  moves  along  T3,  its  projection  QQ 
on  PP  moves  along  T2  with  velocity 

( 9 )  v" =d/dt (QQ) =QQ+QQ=QQ+Qv ' 


along  T2.  Observe  that  v",v',  and  Q  all  lie  in  the  same  plane 


<Note  4>.  This  means,  however,  that  the  following  two  vector 
equation  hold  simultaneously: 

(10)  v'xQ=(v"*Q)/sin(A)  and  v'  .Q=0 

We  can  solve  for  v'  from  these  two  simultaneous  vector  equations 
<Note  6>  to  obtain 

(11)  v *=Qx (v" xq) /sin( A) =csc ( A) v"-cot ( A) Q 

This  is  the  main  equation.  Note  that  v'  is  the  unit  tangent  to 
T3  at  Q.  Its  first  time  derivative,  v',  thus  lies  along  the 
principal  normal  to  T3  at  Q.  Their  vector  product  in  turn  spe¬ 
cifies  the  direction  of  the  sought  angular  velocity  vector  A. 
Differentiating  (11)  we  obtain 

(12)  v'=csc(A)v"-Acsc(A) {cos (A) v"-Q}-cot ( A) v' 

Taking  the  vector  product  of  (12)  with  v'  and  using  the  relation 

v' xv=cos (A) (v ' xq)  leads  to 

*  •  • 

(13)  v'xv"=csc(A) (v'xv")+A (v’xq) 

Substituting  for  v'  from  (11),  for  v'  from  (8),  and  simplifying, 
we  finally  obtain 

(14)  v' xv' =Qcsc ( A) 2 (v"-cos ( A) Q) xv"+QAcsc ( A) (v"xq) 

Now,  A=d/dt(A)  can  be  expressed  in  terms  of  v"  and  the  relation 
between  the  visual  direction  Q  and  the  projection  plane  PP  (its 
unit  normal).  Using  v'.Q=cos(A)  and  differentiating  we  obtain 

(15)  A=-csc (A)  (v" . Q+v" . Q) =-csc  ( A )  (v".Q+v”.v') 

However,  v" . v ' = (v' sin ( A) +Qcos ( A) ) . v ' =v ' sin ( A ) ,  because  v' ,Q=0 
by  definition.  Substituting  for  v'  from  (8)  yields 


(16)  A=-csc(A) (v" .Q) -v"sin(A)/Q 

Equations  (14)  and  (16)  indicate  that  the  direction  of  the 
angular  velocity  vector  at  a  visual  direction  Q  can  be  deter¬ 
mined  completely  once  the  image  velocity  vector  v"  at  the  locus 
on  PP  corresponding  to  the  visual  direction  Q,  and  the  first 
time  derivative  of  the  direction  vector  of  v" ,  are  available. 

The  only  other  quantities  entering  the  equation  are  Q  and  A , 
expressing  the  metrics  of  the  projective  system  <Note  6>. 

Observe  that  the  direction  of  the  vector  v"  is  known  as 
soon  as  v"  is  known.  Because  T2  is  a  planar  motion,  v"  lies 
in  PP,  and  is  perpendicular  to  v" .  Referring  to  Figure  7,  we 
see  that  the  direction  of  v" ,  v"  is  specified  by  v"=cos(n)x  + 
sin(n) y  where  x  and  y  are  a  set  of  mutually  perpendicular  unit 

vectors  on  PP.  v"  is  thus  given  by 
•  • 

(17)  v"=n [-sin(n) x  +  cos(n)y] 

•  • 

and  the  magnitude  of  v”  is  n. 

Finally,  it  remains  to  specify  the  magnitude  of  A.  Observe 
that  A,  the  magnitude  of  A,  is  a  function  of  the  mutual  orienta¬ 
tion  of  A  and  the  direction  vector  Q.  This  is  related  to  the 
already  mentioned  fact  that-v'  and  Q  do  not  specify  A  uniquely 
(see  Figure  8).  Using  v'=Axq  we  see  that  v'sv'v'sAtAxQ) =Asin(w) v 
and  from  this  it  follows  directly  that 

(18)  A=v'/sin(u))=v"sin(A)/[Qsin(w)  ] 

Here,  w  is  the  angle  between  A  and  Q,  and  cos(w)=A.Q. 


Computing  local  surface  orientation 

In  this  section,  we  analyze  a  method  of  computing  local  sur¬ 
face  orientation  relative  to  a  given  visual  direction.  An  inter¬ 
esting  result  will  be  that  the  directions  of  angular  velocity 
vectors  are  not  required  explicitly.  However,  we  cannot  obtain 
something  for  nothing;  the  analysis  requires  an  a  priori  know¬ 
ledge  of  the  (instantaneous)  direction  of  motion  (the  direction 
of  the  translatory  component  of  the  relative  motion) . 

First,  let  us  express  vectors  in  a  cartesian  (rectangular) 
coordinate  frame  as  a  function  of  two  angles  a  and  3.  Then,  for 
a  given  visual  direction  Q(a,3),  we  can  compute  3A/9a  and  9A/93. 
Using  equation  (3) ,  it  is  easy  to  see  that 
(19)  gA/gc^g^/gcx  and  9A/9B=9AT/9g. 

In  other  words,  the  information  contained  in  the  gradient  of  the 
angular  velocity  field  in  the  given  visual  direction  is  equiva¬ 
lent  to  the  information  contained  in  the  gradient  of  its  transla¬ 
tory  component.  This  is  not  surprising,  for  as  we  saw  above,  only 
the  translatory  component  carries  information  about  depth  rela¬ 
tions  between  the  3D  texture  elements. 

Let  us  find  expressions  for  9^/90  and  9^/93  and  analyze 
them  to  see  how  local  surface  orientation  is  specified  in  these 
expressions.  If  a  and  3  are  chosen  such  that  the  vector  corres¬ 
ponding  to  a=0  and  3=0  specifies  the  direction  of  the  translatory 
component  v,  we  see  that  A.^  does  not  change  as  we  move  on  the 
plane  a=const.  Using  this  fact  and  differentiating  (2)  with 
respect  to  3  yields 


(20)  8AT/3e»aAT/ag  at 

•  • 

Now,  8=vsin(3)/S  (see  Figure  2),  and  3AT/3g=9B/9B .  Thus, 

(21)  3B/3tf=vcos (8) /S-BS^/S  where  Sg= 

Multiplying  both  sides  by  tan(B) ,  and  recalling  that  &=vsin(8)/S, 
we  see  that 

tan(B) 30/30=B-Btan ( 3)  Sg/S  so  that 

(22)  Sp/S=-l/8 { 38/38] +cot (3) 

To  derive  a  similar  expression  for  S^/S  requires  a  little 
more  ingenuity.  Differentiating  equation  (2)  with  respect  to  a 
yields 

(23)  3AT/3a=-sin(8) [v/S] [S^/S] AT+ [v/S] (v*Qa) 

But  we  have  also 

(24)  3  At/  3  a*  3  ( A^ )  /  3  a»  3 A?/  3  aAT+ AT  3 AT/  3 a=  3  6/  3  aAT+ X 

where  X  is  some  vector  which  does  not  have  to  be  specified  in 
detail  (for  our  purposes).  It  can  be  seen  immediately  that 

(25)  36/3a=- [Sa/S]v  sin(6)/S 
and  from  this  S^/S  is  specified  as 

(26)  S  /S=-ll/8I36/3a 

The  quantities  S^/S  and  S^/S  are  depth  invariant  characterizations 
of  (local)  surface  orientation  relative  to  a  particular  visual 
direction  Q(a,8)  <Note  8>.  In  fact,  they  are  directly  related 
to  the  gradient  of  the  distance.  Because  of  the  dependence  of 
this  specification  of  local  surface  orientation  on  a  particular 
visual  direction  (defining  the  surfaces  of  constant  a  and  8),  two 
different  surface  orientations  cannot  be  directly  compared.  To 


do  so,  one  could  transform  one  characterization  into  another 

using  a  simple  rotation  matrix.  It  is  important  to  realize  that 

the  expressions  characterizing  the  (local)  surface  orientation 

in  a  pure  translatory  situation  hold  also  in  a  general  situation 

of  a  curvilinear  motion.  The  only  prerequisite  is  (as  in  the 

pure  translation  case)  that  the  direction  of  the  instantaneous 

motion  (i.e.,  the  direction  of  the  translatory  component  of  the 

curvilinear  motion)  is  known.  The  knowledge  of  v  allows  us  to 

define,  for  each  "retinal"  locus,  a  direction  along  which  a=const. 

By  projecting  the  "retinal"  velocity  vector  into  this  direction 

we  obtain  0,  and  by  differentiating  these  "retinal"  velocities 

along  the  directions  a=const.  and  0=const.  (see  Figure  10) 

•  • 

we  obtain  30/3a  and  30/30. 

If  this  process  is  to  be  carried  out  on  the  projection  plane, 
the  best  thing  to  do  is  to  locate  the  focus  of  expansion  which 
then  determines  the  lines  ot=constant  as  the  lines  joining  the 
focus  of  expansion  and  the  particular  "retinal"  locus.  The  locali¬ 
zation  of  the  focus  of  expansion  (FOE)  is  a  difficult  task.  One 
may  try  to  decompose  the  image  velocity  field  there  into  its  con¬ 
stituents.  The  rotational  component  of  the  angular  veloctiy  vector 
at  a  given  instant  can  be  decomposed  into  two  image  velocity  fields 
one,  a  hyperbolic  field  (the  component  vectors  being  tangent  to 
hyperbolas  through  given  loci)  due  to  rotation  about  an  axis 
through  the  center  of  projection  parallel  to  the  projection  plane, 
and  the  other,  a  circular  field  (the  component  vectors  being  tan¬ 
gent  to  circles  with  centers  at  the  center  of  the  image  coordinate 


system)  due  to  rotation  about  the  normal  to  the  projection  plane. 
The  remaining  translational  component  would  consist  of  vectors 
defining  the  focus  of  expansion  (possibly  at  infinity)  as  the 
unique  intersection  of  all  straight  lines  defined  by  these  vec¬ 
tors  and  the  corresponding  "retinal”  loci.  The  whole  process 
is  essentially  a  constrained  minimalization  problem  of  a  func¬ 
tion  of  three  variables:  the  magnitude  of  the  circular  component, 
the  direction  of  the  hyperbolic  field  (specified  by  a  single 
angle) ,  and  the  magnitude  of  the  hyperbolic  field  (see  Figure 
11) .  The  constraining  condition  is  that  the  straight  lines  of 
the  translational  component  meet  at  FOE.  Note  that  such  a  pro¬ 
cess,  if  successful,  would  essentially  recover  the  translational  as 
well  as  the  rotational  component  of  the  angular  velocity  vector  A 
(see  equation  (1)).  We  are  currently  trying  to  solve  this  prob¬ 
lem  using  a  relaxation  scheme  <Note  9>. 


5 .  Conclusion 

In  this  paper  we  have  shown  that  the  relative  depth  of  any 
two  texture  elements  moving  in  the  same  way  relative  to  the  ob¬ 
server  can  be  computed  in  a  simple  way  from  the  angular  velo¬ 
cities  of  the  corresponding  rays  of  projection.  We  have  illus¬ 
trated  how  the  required  angular  velocity  at  a  point  on  the  planar 
projection  surface  can  easily  be  computed  from  the  (linear) 
velocity  of  the  image  element  at  that  locus,  and  the  first  time 
derivative  of  its  direction  vector.  We  have  also  shown  that 
local  surface  orientation  can  be  obtained  rather  straightforwardly 
once  the  direction  of  the  translatory  component  of  the  relative  motion 
is  known.  The  recovery  of  this  direction  from  the  information 
contained  in  the  distribution  of  the  "retinal"  velocities  is  a 
rather  complicated  task.  It  is  hoped  that  it  may  be  possible  to 
decompose  the  instantaneous  velocity  field  on  the  projection  plane 
into  its  constituents  using  a  relaxation  process.  Some  work  on 
this  problem  is  currently  in  progress  in  our  laboratory. 

The  applicability  of  the  method  will  depend  on  the  accuracy 
with  which  the  image  velocities  can  be  obtained.  It  remains  to 
be  specified  how  these  errors  will  propagate  through  the  equations 
and  affect  the  accuracy  of  the  computed  relative  depth  and  local 
surface  orientation. 

The  computation  of  relative  depth  and  local  surface  orienta¬ 
tion  were  presented  as  two  distinct  processes.  This  does  not 
have  to  be  so.  Local  surface  orientation  may  be  obtained  from  a 


relative  depth  map,  for  example,  by  simply  fitting  a  plane  to 
a  set  of  relative  depth  values  in  a  given  (small)  neighborhood. 

I  believe  that  the  relative  depth  map  is  practically  a  much  more 
useful  construct  than  local  surface  orientation.  Because  the 
available  data  are  noisy,  the  computation  of  local  surface  ori¬ 
entation  relying  on  quantities  obtained  in  a  small  neighborhood 
of  a  "retinal"  point  is  likely  to  be  affected  much  more  than 
the  relative  depth  of  two  widely  separated  points,  where  the 
two  angular  velocities  can  be  obtained  much  more  precisely,  e.g. 
by  averaging  over  a  small  neighborhood. 
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Notes 


<Note  1> 

The  following  notation  is  used  throughout  the  paper. 

If  v  is  a  vector  then  v  is  a  unit  vector  in  its  direction,  and  v 
is  the  magnitude  of  v.  The  scalar  product  is  denoted  by  * . " ,  and 
the  vector  product  by  All  velocities,  position  vectors,  and 

the  associated  quantities  are  functions  of  time.  This  is  assumed 
implicitly  throughout  the  paper,  v  and  v  denote  the  time  deri¬ 
vatives,  as  isuai..  Angular  velocities  are  vectors  perpendicular 
to  the  plan'?  cl  (instantaneous)  rotation,  with  magnitude  equal  to 
angular  speed  (radians/sec) . 

<Mote  2> 

The  texture  elements  can  be  on  two  different  objects  as  long  as 
the  objects  move  in  the  same  way  relatively  to  the  observer  (i.e., 
have  the  same  v  and  A^) .  Thus,  for  example,  in  the  stationary 
world  where  the  observer  is  the  only  moving  agent,  the  relative 
depth  of  all  texture  elements  can  be  recovered  using  the  present 
method. 

<Note  3> 

The  structure-f rom-motion  theorem  states  that  the  relative  depth 
of  four  non-coplanar  points  is  recoverable  from  three  non-degenerate 
orthographic  projections.  The  mutual  orientation  of  the  projection 
planes  has  to  be  determined  before  the  actucil  relative  depth  of 
the  four  points  can  be  computed.  The  recovery  of  the  mutual  orienta¬ 
tion  of  the  projection  planes  is  an  integral  part  of  the  schema. 

<Note  4> 

To  see  this,  note  that  v"=d/dt (QQ) =QQ+QQ 

Now  Q=v '  and  v ' . Q=0 .  Thus  v" *Q*Q (v' *Q) ,  i.e.,  v' ,  v",  and  Q  all 
lie  in  the  same  plane.  Setting  n  to  be  the  unit  normal  of  this  plane, 
we  have  v'xQ»v'n;  but  also  v"xQ«v"sin(X) n  (see  Figure  4).  Thus 
n=(v' xQ)/v'*(v"xQ)/v"sin(A) .  Substituting  for  v"xq  we  obtain 

(v‘ xQ)/v'*Q(v'  xQ)/v"sin(X)  and  so  v'*v"sin(X) /Q,  as  stated  in  (8). 


<Note  5> 

A  set  of  vector  equations  of  the  form 

X  x  A=B  and  X . C=p 

where  A.Cj*  0,  has  a  general  solution  X= (pA+C*B) / (A.C) . 

<Note  6> 

Given  the  unit  normal  in  defining  the  projection  plane  PP,  these 
quantities  are  computable  easily  as 

Q=l/ (Q . in)  ,  and  X  is  given  by  cos(A)=(Q.v") . 

<Note  1> 

As  can  be  seen  in  Figure  9,  A^,  is  the  same  for  all  Q(a,B)  on  the 
plane  a=const. ;_it  is  the  unit  normal  of  this  plane.  It  follows 
directly  that  =  0. 

<Note  8> 

Expressions  similar  to  (22)  and  (26)  were  also  independently  ob¬ 
tained  by  Clocksin  (1980)  using  a  different  approach. 

<Note  9> 

While  the  problem  is  conceptually  rather  simple,  there  are  some 
difficulties  relating  to  the  formulation  of  the  criterion  function 
to  be  minimized.  The  difficulty  is  related  to  the  fact  that  the 
projection  plane  is  an  augmented  Euclidean  plane,  in  terms  of  pro¬ 
jective  geometry.  For  example,  the  translational  vector  components 
on  the  plane  may  all  meet  at  an  (ideal)  point  at  infinity.  It  is 
rather  difficult  to  incorporate  this  condition  into  a  nicely  behaving 
criterion  function. 

<Wote  10> 

This  result  is  intuitively  rather  surprising.  It  follows  directly, 
however,  from  the  definition  of  the  angular  velocity  (see  Figure  2) . 


Figure  1 .  A  texture  point  P  projects  into  a  point  ('  on  the  unit 
sphere.  The  direction  vector  of  the  projecting  ray  OP  is  determined 
by  the  two  angles,  a,  the  meridian,  and  3,  the  eccentricity;  the 
vector  Q  is  a  function  of  a  and  g.  The  plane  a=0  and  the  direction 
(a=0,B=0)  are  arbitrary,  but  it  is  advantageous  for  the  future 
analysis  to  choose  them  so  that  the  principal  x-axis  coincides  with 
the  direction  vector  of  the  translatory  motion  component. 


0 


Figure  2.  The  angular  velocity  of  a  ray  due  to  a  pure  translation. 
The  angular  speed  is  defined  as  dg/dt,  i.e.,  as  the  projection  of 
v  on  the  perpendicular  to  the  ray,  divided  by  the  distance  S. 


Figure  3.  To  compute  the  direction  of  the  angular  velocity  vector 
of  the  ray  specified  by  the  visual  direction  observe  that  Q  moves 
( on^a  3D  path  on  the  surface  of  the  unit  sphere)  wit]?  velocity 
v ' =Q .  Because  v'  is  the  unit  tangent  to  this  path,  v  lies  in  the 
direction  of  the  principal  normal  to  the  path  at  Q.  The  unit  bi¬ 
normal  yector  at  which  is  perpendicular  to  the  plane  spanned  by 
v'  and  v' ,  is  parallel  to  the  angular  velocity  vector  A  of  Q. 


Figure  4.  The  basic  projection  geometry.  The  ray,  determined  by 
its  direction  vector  Q,  moves  due  to  the  relative_motion  of  the 
object  with  respect  to  the  observer.  The  point  QQ=Q  at  which  it 
pierces  the  planar  projection  surface,  PP,  describes  a  planar  trajec 
tory  T2.  The  unit  vector  Q  describes  a  3D  trajectory  T3.  The  angle 
A  is  the  angle  between  the  image  velocity  v"  of  Q,  the  projection 
of  P  onto  PP,  and  the  ray  OP. 


Figure  5.  The  radial  projection  equation.  The  displacement  v  on 
i  at  unit  distance  from  the  center  of  projection  is  projected  into 
tiie  displacement  v"  on  l~.  To  compute  the  relation  between  v 
and  v”,  we  note  that  b/(a+Q)=V;  b/v”  =  sin ( X)  ;  and  a/v  =cos(X)  Thus 
v” sin ( X) =v ' v"cos ( X) +v'Q.  Finally,  v"=v'Q/ (sin(X) -v  cos(X)).  Ob¬ 
serve  that  here,  v*  and  v"  are  finite  displacements  and  not  velo¬ 
city  vectors! 


Figure  6.  The  relation  between  an  infinitesimal  displacement  d<P 
along  the  circle  C,  and  its  projection  on  the  line  The  speed 

at  which  b,  the  projection  of  a  on  moves  along  1 

not  the  projection  of  the  speed  with  which  a  moves  along  C. 
See  text  for  further  explanation. 
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Figure  7.  The  direction  of  on  PP  is  determined  by  an  angle  n. 
The  first  time  derivative  of  v"  has  direction  perpendicular  to  v", 
and  magnitude  n.  x  and  y  are  a  set  of  mutually  perpendicular  unit 
vectors  on  the  projection  plane  PP . 


Figure  8.  Knowing  that  the  point  Q  (with  position  vector  Q)  moves 
with  some  (linear)  velocity  v  does  not  specify  the  angular  velocity 
of  the  ray  OQ.  The  equation  v=A*Q  constrains  A  to  lie  in  the  plane 
of  which  v  is  a  normal.  Q  could  (instantaneously)  move  on  an  infinite 
number  of  possible  circles  of  rotation,  only  three  of  them  being 
shown.  Observe  that  A,  the_magnitude  of  A,  depends 
on  the  angle  between  Q  and  A  (it  determines  the  radius  of  the  in¬ 
stantaneous  circle  of  rotation) . 
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c,  c, 


Figure  10.  For  each  given  visual  direction  Q(a,B),  the  circles 
u=const.,  (3=const.,  and  <J  itself  define  a  rectangular  coordinate 
system.  Observe  that  while  the  condition  a=const.  defines  a  plane, 
3=const.  defines  a  cone  with  apex  at  O'  The  circles  on  the 
plane  and  C 2  on  the  cone  are  mutually  perpendicular. 


Figure  11.  An  image  velocity  v"  (on  the  planar  projection  sur¬ 
face  PP)  of  a  point  Q  can  be  resolved  into  three  components.  The 
hyperbolic  component  h  is  due  to  the  rotation  of  the  ray  about  an 
axis  (through  0)  in  the  projection  plane  (the  angular  velocity 
is  a  linear  combination  of  x  and  y) .  The  circular  component  c 
is  due  to  the  rotation  of  the  ray  about  an  axis  (through  o)  paral¬ 
lel  to  m.  The  translational  component  t  is  the  remaining  vector 
which  constraints  the  decomposition  of  v" ;  t  is  constrained  to  be 
such  that  VQ.fcPP:  (t.  intersect  in  one  common  point).  In  the 
illustration ^above,  the  direction  angle  of  the  hyperbolic  field 
is  zero  (measured  anticlockwise  from  the  x-axis) . 
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the  projection  surface  and  the  first  time  derivative  of  its 
direction  vector.  A  similar  analysis  produces  a  set  of  equa¬ 
tions  which  directly  yield  local  surface  orientation  relative 
to  a  given  visual  direction.  The  variables  involved  are  scalar 
quantities  directly  measurable  on  the  projection  surface  but, 
unlike  the  case  of  relative  depth,  the  direction  of  (instan¬ 
taneous)  motion  has  to  be  computed  by  different  means  before 
the  method  can  be  applied.  The  relative  merits  of  the  two 
formalisms  are  briefly  discussed. 


